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Abstract 

Most recent software related accidents have been sys- 
tem accidents. To validate the absence of system hazards 
concerning dysfunctional interactions, industrials call for 
approaches of modeling system safety requirements and 
interaction constraints among components and with envi- 
ronments (e.g., between humans and machines). This paper 
proposes a framework based on input/output constraint 
meta- automata, which restricts system behavior at the meta 
level. This approach can formally model safe interactions 
between a system and its environment or among its com- 
ponents. This framework differs from the framework of the 
traditional model checking. It explicitly separates the tasks 
of product engineers and safety engineers, and provides 
a top-down technique for modeling a system with safety 
constraints, and for automatically composing a safe system 
that conforms to safety requirements. The contributions of 
this work include formalizing system safety requirements and 
a way of automatically ensuring system safety. 

1. System Accidents 

Computer technology has created a quiet revolution in 
most fields of engineering, also introduced new failure 
modes that are changing the nature of accidents (TJ. To 
provide much more complex automated services, industrials 
have to develop much more complicated computer systems 
which consist of numerous components and a huge number 
of actions (both internal and interactive). A recent challenge 
is the system accident, caused by increasing coupling among 
system components (software, control system, electrome- 
chanical and human), and their interactive complexity 0(3]. 
In contrast, accidents arising from component failures are 
termed component failure accidents. 

System safety and component reliability are different. 
They are system property and component property, respec- 
tively [2|. Reliability is defined as the probability that a 
component satisfies its specified behavioral requirements, 
whereas safety is defined as the absence of accidents — 
events involving an unacceptable loss [4|. People are now 
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constructing intellectually unmanageable software systems 
that go beyond human cognitive limits, and this allows 
potentially unsafe interactions to be undetected. Accidents 
often result from hazardous interactions among perfectly 
functioning components. 

There are several examples of system accidents. 

The space shuttle Challenger accident was due to the 
release of hot propellant gases from a field joint. An O- 
ring was used to control the hazard by sealing a tiny gap in 
the field joint created by pressure at ignition. However, the 
design did not effectively impose the required constraints 
on the propellant gas release (i.e., it did not adequately seal 
the gap), leading to an explosion and the loss of the shuttle 
and its crew Q5J. 

The self-destructing explosion of Ariane 5 launcher was 
resulted from the successive failures of the active inertial 
reference system (IRS) and the backup IRS [5|. Ariane 5 
adopted the same reference system as Ariane 4. However, 
the profile of Ariane 5 was different from that of Ariane 4 
— the acceleration communicated as input value to IRS of 
Ariane 5 was higher. Furthermore, the interactions between 
IRS and other components were not checked and redefined. 
Due to the overflow of input value computation, the IRS 
stopped working [6 1. Then, the signaled error was interpreted 
as a launcher attitude, and led the control system to rotate 
the tailpipe at the end stop Q. 

The loss of the Mars Polar Lander was attributed to a 
misapprehensive interaction between the onboard software 
and the landing leg system |8|. The landing leg system was 
expected and specified to generate noise (spurious signals) 
when the landing legs were deployed during descent. How- 
ever, the onboard software interpreted these signals as an 
indication that landing occurred (as specified in their require- 
ments) and shut the engines down, causing the spacecraft to 
crash into the Mars surface. 

A system accident occurred in a batch chemical reactor 
in England [9|. The computer controlled the input flow of 
cooling water into the condenser and the input flow of 
catalyst into the reactor by manipulating the valves. The 
computer was told that if any component in the plant gets 
abnormal, it had to leave all controlled variables as they were 
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Figure 1 . A Chemical Reactor Design 



and to sound an alarm. On one occasion, the computer just 
started to increase the cooling water flow, after a catalyst had 
been added into the reactor. Then the computer received an 
abnormal signal indicating a low oil level in a gearbox, and 
it reacted as its requirements specified: sounded an alarm 
and maintained all the control variables with their present 
condition. Since the water flow was kept at a low rate, 
then the reactor overheated, the relief valve lifted and the 
contents of the reactor were discharged into the atmosphere. 
The design of the system is shown in Fig. Q] 

In all these accidents, the components are reliable in 
terms of satisfying their specified requirements, but the 
systems are not safe as a whole. As Leveson mentioned in 
STAMP (Systems-Theoretic Accident Model and Processes) 
HI, these accidents result from inadequate control or en- 
forcement of safety- related constraints of the systems. 

Since most software related accidents have been system 
accidents [1|, people need to model and constrain inter- 
actions of system components to validate the absence of 
dysfunctional interactions. One of the challenges to system 
safety is that the ambiguity in safety requirements may lead 
to reliable unsafe systems. Another challenge is the lack of 
formal techniques for describing safety rules and interac- 
tions between components (including interactions between 
humans and machines), which makes automated verifications 
difficult. In this paper, we will propose a formal framework 
for modeling system safety rules. The framework is based 
on a new concept of I/O constraint meta-automata. 

This paper is organized as follows: the framework of our 
approach is proposed in Section 2. The formal technique 
based on I/O constraint meta-automata is introduced in 
Section 3. An example is used to illustrate how to formalize 
safety constraints and combine it with a system specification. 
In Section 4, we discuss how to apply this approach to 
a system that consists of multiple components. In Section 
5, we compare our works to classic verification techniques, 
such as model checking, and conclude the paper. 



2. The Framework of Our Approach 

The most popular technique of system safety verification 
is model checking flOl . In this framework, we have two 
steps in verifying a system. At first, we formalize system 
behavior as a model (e.g., a transition system, a Kripke 
model ifTTI ). At the second step, we specify the features that 
we aim at validating, and use a certain checking algorithm 
to search for a counterexample which is an execution trace 
violating the specified features. If the algorithm finds such 
a counterexample, we have to modify the original system to 
ensure safety requirements, or else the verification succeeds. 

Unlike the model checking, our framework takes another 
way. It consists of the following steps: 

1) Modeling system behavior, including specifications of 
its components, internal and external interactions. 

2) Modeling system safety constraints using a certain 
formal technique, e.g., I/O constraint meta-automata 
in this paper. 

3) Combining these two models to deduce a safe system 
model, that is, a system model whose behavior is in 
accordance with its safety constraints. 

As we mentioned in |[T2l . the system behavior specifies 
an operational semantics, which defines what a system is 
able to do. Modeling system behavior is mainly performed 
by product engineers (designers), such as programmers and 
developers. In the example of the chemical reactor control 
system, the actions "opening the catalyst flow", "opening 
the cooling water flow" and "sounding an alarm" are system 
behaviors. 

In the second step, the model of safety requirements 
specifies a correctness semantics, which defines what a 
system is authorized to do. This process is the duty of safety 
engineers whose responsibility is to assure system safety. 
Safety engineers may consist of requirement engineers, test 
engineers, managers from higher socio-technical levels who 
define safety standards or regulations (TJ, etc. In the example 
of the chemical reactor system, the constraint "opening the 
catalyst flow must be followed by opening the cooling water 
flow" is an instance of system safety requirements. 

In the third step, in order to ensure system safety, we 
combine the system model with its safety constraints model. 
Then we can check if the system is safe under the constraints 
specifying safety requirements. However, the precondition 
of such a formal checking is that we must formalize safety 
requirements. And we also need to carefully define the 
composition of a system model and its constraints model. In 
the next section, we will introduce such an approach based 
on I/O constraint meta-automata. 

We remark here that an other precondition is that we can 
find all safety constraints in a system. However, this is an 
issue of "risk identification" fl3l , which is outside the scope 
of this paper. This work deals with "risk treatment", which 
is a different phase to risk identification, according to the 
ISO Standard 31000 El. 



3. Modeling System Safety Constraints Using 
I/O Constraint Meta-Automata 

The theory of input/output automata |14]|15| extends 
classic automata theory [16| for modeling concurrent sys- 
tems with different input, output and internal actions. I/O 
automata and the variants are widely used in modeling 
distributed systems ifTTl . 

Definition 1: An input/output automaton (also called 
an I/O automaton or simply an automaton) is a tuple 
A = (Q, E 7 , E°, E H , 5, S), where: 

« Q is a set of states. 

• E 7 , E° ,T, H are pairwise disjoint sets of input, out- 
put and internal actions, respectively. Let E = 
E 7 U E° U T, H be the set of actions. 

• 5 C Q x E x Q is a set of labeled transitions, such 
that for each a € E 7 and q e Q there is a transition 
Pfc : (q,a,q') e <5 (input-enabled). 

• <S C Q is a nonempty set of start states. □ 
In the graph notation, a transition p^ : (q, a, q') is denoted 

by an arc from q to q' labeled p^ : a, where p^ is the name 
of the transition. To discriminate explicitly the different sets 
of actions in diagrams, we may suffix a symbol "?", "!" or 
";" to an input, output or internal action, respectively. 

In the example of the batch chemical reactor, the computer 
system behavior is modeled using an I/O automaton A 
of Fig. I2l). The automaton A includes a set of input 
actions E 7 = {1} (low oil signal), a set of output actions 
E° = {c,w,a} (opening catalyst flow, opening water flow, 
sounding an alarm, respectively), and a set of internal actions 
E 77 = {e} (ending all operations). The normal operational 
behavior includes opening catalyst flow (pi), then opening 
water flow (P2), etc., resulting in an infinite execution trace 
PiP2PiP2---- To respond to abnormal signals as soon as 
possible, all the states have a transition labeled I, which 
leads to a state that can sound an alarm (pe) and stop the 
process (ps). Unfortunately, this design leads to hazardous 
behaviors: (cw)*clae, that is, after a sequence of opening 
catalyst and water flows (cw)*, then the catalyst flow is 
opened c when an abnormal signal is received I, then an 
alarm is sounded a. So water is not added after the catalyst 
flow is opened. This sequence of events leads to the accident 
mentioned in Section 1. 

Note that this hazard is due to the uncontrolled sequences 
of transitions — p\ must be followed by p2 and not by p^. 
To solve this problem, we need to specify the authorized 
sequences (satisfying safety constraints) on the transitions 
S and not on the actions E. Thus, these constraints are 
not at the behavioral model level, but at the meta-model 
level. We propose the concept of constraint meta- automata 
to formalize safety constraints. Then, we combine a meta- 
automaton with the system automaton. 

Definition 2: A constraint meta-automaton (or simply 



meta-automaton) A over an I/O automaton A = (Q, E, 5, S) 
is a tuple A = (Q, E, 5, S), where: 

• Q is a set of states disjoint with Q. 

• E is a set of terminals that consists of all the transition 
names in S of A. 

• 5 is a set of labeled transitions. 

• S C Q is a nonempty set of start states. □ 
Note that the transitions 5 of A are terminals of A, so we 

say that A is at the meta level of A. Figure[3]illustrates the 3 
levels in our framework. Let E* be a set of execution traces 
of actions, A describes the behavior on E. A specifies the 
behavior on the ^4-transitions (E = 5), that is, a behavior 
on the behavior of A. This meta-behavior expresses safety 
requirements. 
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Figure 3. A 3-levels Overview 

In the example, to prevent accidents, we need to bind 
the safety constraint "opening catalyst must be followed 
by opening water," that is, "whenever the transition p\ : c 
occurs, the transition p2 : w must occur after that." This 
constraint can be formalized as a constraint meta-automaton 
A of Fig. |2j2). When we design this constraint, we only 
specify the sequence of transitions p\ , p2 at the meta-model 
level, and we concern little about the implementation of the 
system at the model level. The next step is to compose the 
system automaton A with its constraint meta-automaton A, 
and automatically generate a system model A' satisfying the 
safety requirement. 

Definition 3: The meta-composition A' of an I/O au- 
tomaton A = (Q, E, 6, S) and a constraint meta-automaton 
A = (Q, E, 5, S) over A is a tuple: 



A' = A^A ={QxQ, E, 5', SxS) 



(1) 



where p k : ((qi,qj),a, (q m , q n )) e 5' iff, 

(1) Pk ■ (ft, a, q m \£ S, and 

(2) (q 3 ,Pk,q n ) S 6. □ 
The symbol • is the meta-composition operator, and 

read as "meta-compose". Its left and right operands are an 
automaton and a constraint meta-automaton, respectively. 

Notice that S = {pk}keic plays a key role in associating 
transitions of A and terminals of A. For our example, we 
combine the automata A and A of Fig. [2] thus we get the 
automaton A' = A^A of Fig. [4] where denotes (qi, qj). 
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Figure 2. Automata of the Reactor Control System 
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Figure 4. The Meta-Composition A' 



The meta-composition contains exactly all the paths sat- 
isfying the constraint in the system. Formally, we have the 
following theorem (the proof is omitted for its simpleness 
and intuitiveness from the definition): 

Theorem 4: Given A, A and the meta-composition A', an 
execution trace is e E* is in A' iff, is is in A, and its 
transition trace i d e 8* is in A. □ 

Obviously, the set of traces of A' is a subset of the traces 
of A. Formally, let L(A) be the set of traces of A (also the 
language of A), we have L(A') C 

Thanks to A, the hazardous execution traces, for example 
cwclae, which exists in A, will be eliminated, because its 
transition trace piP2PiPiP&Pd, & L(A) (the language of A). 
The comparison between A of Fig. 01) and A 1 of Fig. g] 
highlights the hazardous transition p4 of A. However, in 
general, this diagnosis is much more complex and cannot be 
achieved manually, since a real system A has too many states 
to be expressed clearly on a paper. That is why we should 
provide a formal and automated method for eliminating 
hazardous transitions. 

4. Modeling Multi-Component Systems with 
Safety Constraints 

Our approach can also be applied to the systems that are 
made up of several components, whose safety constraints 
are related to several components. 

As a preliminary, we redefine the composition of I/O 
automata, which was introduced in lfl4ll . 

Let M — {ni, nk] C N be a finite set with cardinality 
k, and for each n 6 JV, S n be a set. Then we define: 



S nj )AJ\f = {ni,...,n fe } A (Vji,j 2 G {1, ...,*}• ji < 
32 — ► %i < w j 2 )}- We define the function of projection 
s [j] to denote the j-th component of the state vector s : 
Vj G {1, . .. , fc}, (a; ni , x n2 , x nk ) [j'j = x nj . 

Definition 5: A finite collection of I/O automata 
{A n } nG tf is said to be strongly compatible if Vi, j £ A/", 
i 7^ .7, we have 



(1) E^ns 



and 



(2) Ef n Ej =0. □ 
Definition 6: The composition v4 = IlneA^ ^ n °^ a fi n i te 

collection of strongly compatible I/O automata {A n } ne ^f 

is an I/O automaton (IlneA/ - 

iff, 
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5, 



def 
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UneTV ' 

• S ff = Un§jvj£ . ^d 
. for each q ,~q' e Yl n& x Qn and a G E, 
pi : ("g , a, ') e <5 iff Vj : 1 < j < \J\f\ A rij e JV, 

1) if a € E„. then 

3i :i CI.p, : ("g [j], a, if' [j]) G 5„ 3 ; 

2) if a ^ E nj then ^[7] = If' [7] and 

Vi : p t G <5„ 3 • i n X = 0. □ 
Notice that the name of a transition of A may contain a 
set of names of original transitions px = {pi}icx, where i 
may be a set or a single element. 

We use an example derived from lfl4l . concerning a 
system composed of two components with interactions: a 
candy vending machine and a customer. The candy ma- 
chine A m , specified in Fig. EJl), may receive inputs 61,62 
indicating that buttons 1 and 2 are pushed, respectively. 
It may output s, a, indicating candy dispensation actions, 
SKYBARs and ALMONDJOYs, respectively. The machine 
may receive several inputs before delivering a candy. A 
greedy user A u , specified in Fig. |5j2), can push buttons 
61, 62 or get a candy s, a. The greedy user does not wait for 
a candy bar before pressing a button again. 

The composition of the machine behavior and the user 
behavior is defined by A rnu — A m ■ A u of Fig. 03), where 
qij denotes the composite state (rrii,Uj), Pi lt ...,i k denotes 
a set of transitions {pi 1( Pi 2 , ■ ■ ■ ,Pi k }- A transition of the 
composition may be composed of several transitions of com- 
ponents. For example, 7^1,15 : s is a synchronization of p\ : s! 



and pis : s?, which belong to A m and A u , respectively. 
Formally, a transition of A = YineAf ^ n ma ^ ^ e com P osec l 
of i transitions of components, where 1 < i < \J\f\. 

In the context of composite transitions, a composite tran- 
sition is allowed iff one of its sub-transitions is authorized 
by its constraint meta-automaton. Thus, we define the meta- 
composition operator as follows: 

Definition 7: The meta-composition A' of a composition 
A = n„ eA A A « = (Un&AfQn^^^^Sn) and a 
constraint meta-automaton A = (Q, E, S, S) over A is a 
tuple: 

A' = A^A = ((J[ Q n ) x Q, E, 8', ( J[ S n ) x S) (2) 

nGTV riGTV 

where p x : ((qt, qj),a, (qZ, q n )) E 5' iff, 

(1) Px ■ (Oi ,a,q^i) S 5, and 

(2) 3fc : k G !• (q 3 ,Pk,q n ) 6 <5. □ 
Notice that the specification of the example allows a haz- 
ardous situation: the greedy user repeatedly pushes a single 
button without giving the machine a chance to dispense 
a candy bar (the transition labeled ^5,13 : b\ of qn does 
not allow the transition (qn, s, goo) to be fired). To prevent 
this situation, the following constraints forbid successive 
occurrences of pressing a single button: 

> Whenever one of the transitions Ps,P5,P7 (action b\) 
occurs, the next transition must be not P3,P5,P7. 

* Whenever one of the transitions Pa,Pq,P$, (action 62) 
occurs, the next transition must be not p4,p&,pg. 

Differing from the previous example, the constraint needs 
to synchronize the actions of the machine and of the user. 

Formalizing the constraints, the semantics of the con- 
straint meta-automaton A c of Fig. [6f 1) is: whenever the user 
pushes a button, she or he cannot push it again, but may push 
the other button to change the choice, or wait for a candy 
bar. 

Combining the whole system A mu with its constraint 
A c , we get the system A' — (A m ■ A u )~^ A c in Fig. |6j2), 
where q^ denotes the composite state (nii,Uj, Ck)- All of 
its execution traces satisfy the constraint, and thus prevent 
the hazardous situation. 

This simple example is good for demonstrating the prin- 
ciple, avoiding indigestible diagrams of automata. Since we 
formally defined the meta-composition operator, it can be 
easily implemented to be an automated tool. Thus, it can be 
applied to more complex systems. 

5. Conclusion 

We propose modeling system safety requirements for- 
mally using I/O constraint meta-automata. As we illustrated 
using the examples, this approach can formally model safe 
interactions between a system and its environments, or 
among its components. This framework differs from the 



one of the traditional model checking. It explicitly separates 
the tasks of product engineers and safety engineers, and 
provides a technique for modeling a system with safety 
constraints, and for automatically composing a safe system 
that conforms to safety requirements. 

The essential ideas of our approach are the separation 
and formalization of the system specification A (behavioral 
requirements) and the safety constraints A (safety require- 
ments). The automaton A handles inputs to produce outputs 
using activities depending on the states, whereas the meta- 
automaton A treats activities to produce the set of acceptable 
activities depending on safety requirements. 

Our framework has different objective and uses different 
approaches to those of model checking. Model checking 
techniques use a bottom-up approach — it verifies execution 
traces E* at the lower level L\ to prove the correctness and 
safety of the system model A at the middle level L2 (see 
Fig. [3}. However, our proposal uses a top-down approach 
— we model safety requirements as acceptable sequences of 
transitions (<5*) at the higher level L3 to ensure the correct 
use of A. Then any execution trace (at Li) that conforms 
to the meta-composition A' is definitely a safe execution. 
So the two techniques are complementary. Model checking 
may be used to reduce the fault likelihood, and our approach 
can be applied to avoid behavior that are not in accordance 
with some critical safety requirements. 

Both linear time logic and branching time logic have been 
proved to be useful in checking properties of traces of classic 
automata ifTUl . Since I/O automata are extended from classic 
automata, these existing techniques can be easily applied to 
I/O automata with little modifications. 

In the future, we will apply the approach to the variants 
of I/O automata, e.g., timed, hybrid, probabilistic, dynamic 
ifTTl . Our approach can also be applied to the systems 
specified using classic automata |16|, since I/O automata 
are specific extensions of traditional automata. 

We will also study the formalization of parameterized con- 
straints. To model parameterized systems more accurately, 
parameters of actions and value domains of variables should 
be considered. This is also the basis of studying reusability, 
substitutability and equivalence of components. 

As we mentioned, identification of potential hazards is 
also a challenge in practice. Risk identification and treatment 
are both important phases in risk management [ 13 1. This will 
be a good direction for future work. 
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